feat: implement JIT component experiment #30049

AtofStryker · 2024-08-16T01:32:23Z

Closes Add option for Component Testing to JIT compile test before executing. #29244

Additional details

Some historical context:

Earlier this year, users were and still are experiencing issues with Chunk Load Errors in #28644 where different aspects of the webpack bundle, hosted by webpack-dev-server, are timing out or failing to load. We believe this is due to high memory usage consumed by the dev server due to Cypress compiling all user assets at once needed to run every single component test. This means the dev servers scales with the size of the component testing project, which is not good ☹️ . Because of this, users have been running into the following:

Out Of Memory (OOM) errors.
chunk load error, specific to Webpack where parts of the bundle fail to load.
long run times while running component tests in CI in cypress run mode.
Unable to efficiently develop tests in cypress open mode because the initial and recompiling times for specs and related components is extremely long.
Inability to leverage Smart orchestration effectively to scale resources due to all assets being compiled into the dev server regardless of what specs are run.

The proposed solution to deal with these problems is #29244, a Just-In-Time compilation option that only compiles the assets needed for a given spec right before the spec is run (hence the name just-in-time), with the idea reducing resources needed to run the dev server. That is exactly what this feature is, experimentalJustInTimeCompile.

experimentalJustInTimeCompile

Enabling experimentalJustInTimeCompile will only compile assets and resources related to the spec before the spec is run. This is applicable in both open and run modes.

In open and run modes, the dev server will start up when the Cypress application starts with the component testing type, but will only compile the spec once it is selected or run. When a different spec is run, the different spec is then compiled, omitting the previously run spec from the dev server.

This makes developing your current spec much faster in open mode when it comes to initial compiling and recompiling, with a minor tradeoff being to recompile when you switch specs.

This also significantly reduces the memory requirements needed to run your tests in run mode, with a tradeoff being that your tests might run slightly slower since the dev server needs to recompile. If time is a factor, Smart orchestration can now be effectively used to parallelize your component testing runs on smaller infrastructure since the dev server only compiles what is needed to run the spec.

Steps to test

Since we need a large project to test against, most of the benchmarking was done against internal Cypress repositories. From our experimentation, we saw memory usage drop about 40% with run times being close to the same maybe a little longer for run mode. For open mode, we saw initial compile times reduced by about 60%.

If testing in the cypress repository, there are system tests and cypress-in-cypress tests added to verify both run and open mode. The system tests make sure the dev server is only started once, but the entries are updated as the run progresses through the specs. All specs should be passing for both webpack and vite. The cy-in-cy tests verify that the user can switch between specs in the Cypress app and that the specs pass. If you have the debug flags enabled for cypress:webpack* or cypress:vite*, you will see the entry update to the spec that is selected.

How has the user experience changed?

Users can hopefully leverage this experiment to unburden themselves of the above errors when running their tests in CI and increasing developer productivity with open mode via not having to wait significantly long periods of time for tests to compile.

PR Tasks

Have tests been added/updated?
Has a PR for user-facing changes been opened in cypress-documentation? adds experimentalJustInTimeCompile documentation for Component Testing cypress-documentation#5897
Have API changes been updated in the type definitions?

cypress · 2024-08-16T02:12:17Z

cypress Run #56703

Run Properties: Passed #56703 • 2d29ed1057: Merge branch 'feat/experimentalJustInTimeCompile' of github.com:cypress-io/cypre...

Project	`cypress`
Branch Review	`feat/experimentalJustInTimeCompile`
Run status	`Passed #56703`
Run duration	`17m 35s`
Commit	`2d29ed1057: Merge branch 'feat/experimentalJustInTimeCompile' of github.com:cypress-io/cypre...`
Committer	`AtofStryker`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`1`
Pending	`81`
Skipped	`0`
Passing	`5797`
View all changes introduced in this branch ↗︎

UI Coverage `66.28%`
Untested elements	`27`
Tested elements	`57`

Accessibility `96.6%`
Failed rules	`0 critical` `6 serious` `1 moderate` `0 minor`
Failed elements	`202`

jennifer-shehane · 2024-08-16T14:49:38Z

system-tests/__snapshots__/results_spec.ts.js

@@ -27,6 +27,7 @@ exports['module api and after:run results'] = `
    "experimentalMemoryManagement": false,
    "experimentalModifyObstructiveThirdPartyCode": false,
    "experimentalSkipDomainInjection": null,
+    "experimentalJustInTimeCompile": false,


I approve this change to this results_spec.ts.js file, as a code owner.

…n ci]

cli/types/cypress.d.ts

Co-authored-by: Ryan Manuel <[email protected]>

system-tests/projects/experimental-JIT/vite/cypress/support/component-index.html

system-tests/projects/experimental-JIT/webpack/cypress/support/component-index.html

ryanthemanuel · 2024-08-20T19:46:58Z

npm/webpack-dev-server/src/makeDefaultWebpackConfig.ts

@@ -98,7 +99,8 @@ export function makeCypressWebpackConfig (
    devtool: 'inline-source-map',
  } as any

-  if (isRunMode) {
+  // if experimentalJustInTimeCompile is configured, we need to watch for file changes as the spec entries are going to be updated per test


Do we need to watch everything? Or is there a smaller subset of files that we can watch in the JIT case?

It's actually doing the later. The watching only applies to files that are in the entry, which is the support file, index file and related files, plus the spec entry. I contemplated narrowing this down to just the specpattern, but we have this weird hack where we save the index file with no changes to prompt the dev server to trigger a refresh. I wonder if we can just do the index html file and we get the same effect? Ill try it

@ryanthemanuel So the watch options for webpack are a bit complicated, but I figure a quick win here is to at least ignore the node_modules directory. Vite ignores this by default. updated watch options in 77b5dee

…/component-index.html Co-authored-by: Ryan Manuel <[email protected]>

…mponent-index.html Co-authored-by: Ryan Manuel <[email protected]>

cacieprins · 2024-08-21T16:17:27Z

packages/data-context/src/data/ProjectLifecycleManager.ts

          if (!devServerOptions?.port) {
            throw getError('CONFIG_FILE_DEV_SERVER_INVALID_RETURN', devServerOptions)
          }

          finalConfig.baseUrl = `http://localhost:${devServerOptions?.port}`
+
+          span?.end()


if line 264 gets hit, this span doesn't end - what effect does an orphan span have on the telemetry for dataContext:ct:startDevServer ?

the spans look super messy. Ill fix this. Thank you for catching this!

addressed in e4eaaf8

…ss-io/cypress into feat/experimentalJustInTimeCompile

cypress-bot · 2024-08-27T17:53:31Z

Released in 13.14.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.14.0, please open a new issue.

AtofStryker changed the title ~~implement JIT component experiment~~ feat: implement JIT component experiment Aug 16, 2024

AtofStryker force-pushed the feat/experimentalJustInTimeCompile branch from 4871267 to 42cc2c9 Compare August 16, 2024 14:34

feat: implement JIT component experiment [run ci]

cd55330

AtofStryker force-pushed the feat/experimentalJustInTimeCompile branch from 42cc2c9 to cd55330 Compare August 16, 2024 14:35

jennifer-shehane reviewed Aug 16, 2024

View reviewed changes

AtofStryker requested review from cacieprins and ryanthemanuel August 16, 2024 17:20

AtofStryker marked this pull request as ready for review August 16, 2024 17:22

AtofStryker requested a review from brian-mann as a code owner August 16, 2024 17:22

AtofStryker added 2 commits August 20, 2024 13:06

don't update the dev server unnecessarily if the run is cancelled [ru…

9d4900d

…n ci]

fix system test flake

31b8610

ryanthemanuel reviewed Aug 20, 2024

View reviewed changes

cli/types/cypress.d.ts Outdated Show resolved Hide resolved

Update cli/types/cypress.d.ts

72df233

Co-authored-by: Ryan Manuel <[email protected]>

AtofStryker requested a review from ryanthemanuel August 20, 2024 19:11

ryanthemanuel reviewed Aug 20, 2024

View reviewed changes

system-tests/projects/experimental-JIT/vite/cypress/support/component-index.html Outdated Show resolved Hide resolved

ryanthemanuel reviewed Aug 20, 2024

View reviewed changes

system-tests/projects/experimental-JIT/webpack/cypress/support/component-index.html Outdated Show resolved Hide resolved

ryanthemanuel reviewed Aug 20, 2024

View reviewed changes

AtofStryker and others added 2 commits August 20, 2024 16:24

Update system-tests/projects/experimental-JIT/webpack/cypress/support…

30d7820

…/component-index.html Co-authored-by: Ryan Manuel <[email protected]>

Update system-tests/projects/experimental-JIT/vite/cypress/support/co…

fc30c45

…mponent-index.html Co-authored-by: Ryan Manuel <[email protected]>

cacieprins approved these changes Aug 21, 2024

View reviewed changes

AtofStryker added 3 commits August 21, 2024 17:32

chore: ignore node_modules for webpack dev server

77b5dee

be sure to end span when throwing error

e4eaaf8

Merge branch 'feat/experimentalJustInTimeCompile' of github.com:cypre…

2d29ed1

…ss-io/cypress into feat/experimentalJustInTimeCompile

AtofStryker requested a review from ryanthemanuel August 21, 2024 21:35

ryanthemanuel approved these changes Aug 22, 2024

View reviewed changes

jennifer-shehane approved these changes Aug 22, 2024

View reviewed changes

AtofStryker merged commit 57f6110 into develop Aug 22, 2024
126 of 130 checks passed

AtofStryker deleted the feat/experimentalJustInTimeCompile branch August 22, 2024 14:56

cypress-bot bot locked as resolved and limited conversation to collaborators Aug 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement JIT component experiment #30049

feat: implement JIT component experiment #30049

AtofStryker commented Aug 16, 2024 •

edited

Loading

cypress bot commented Aug 16, 2024 •

edited

Loading

jennifer-shehane Aug 16, 2024

ryanthemanuel Aug 20, 2024

AtofStryker Aug 20, 2024

AtofStryker Aug 21, 2024

cacieprins Aug 21, 2024

AtofStryker Aug 21, 2024

AtofStryker Aug 21, 2024

cypress-bot bot commented Aug 27, 2024

feat: implement JIT component experiment #30049

feat: implement JIT component experiment #30049

Conversation

AtofStryker commented Aug 16, 2024 • edited Loading

Additional details

Some historical context:

experimentalJustInTimeCompile

Steps to test

How has the user experience changed?

PR Tasks

cypress bot commented Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cypress-bot bot commented Aug 27, 2024

AtofStryker commented Aug 16, 2024 •

edited

Loading

cypress bot commented Aug 16, 2024 •

edited

Loading